This is a good start, but I don't think it's complete enough or shows the right metrics for a security analyst -

The total request/response/network bytes aren't that useful without time context or comparative baselines.

Same feedback for "Events by response status" and "Events by privacy status" - these are basically saying "everything worked" but don't give actionable info.

Also, re the top source IP datatable - it doesn't show which processes are communicating, destination IPs, protocols/ports or time of activity which are more relevant...

Can we try editing some of this @muskan-agarwal26 @piyush-elastic - can you let me know if any of these visualizations are possible based on the information available in the logs?

• "Network connections over time" (maybe an area chart?) showing connections/minute, color-coded by new/established/closed
• "Active network connections" table showing process, local port, remote IP, state, duration
• "Top external destinations" table showing domains/IPs and # of connections

cc @jamiehynds

Sure, @cpascale43 , I’ll follow your suggestions and make the necessary changes.

Similar feedback to the Network dashboard - can we make sure this has more security context?

The biggest enhancements would be an events timeline, and a more security-specific events breakdown table.

The pie charts up top aren't very meaningful on their own, so I think we could replace them with an area chart showing aggregated security events over time. Is there a way a user could see key events represented on the top and when they occurred, like

• "3 failed authentication attempts at 12:23"
• "New process launched from /tmp at 9:15"

Also, the "Events by Subsystem" and "Events by Category" bar charts are a bit strange - I think it would be more useful to bucket the events into categories like "Authentication", "Process", "Network", "File system" etc, in line with the categories outlined in the issue.

Can we make a "Security Event Types" table showing a breakdown of the number of events per category? Something like

Authentication Events
---------
- Successful logins | 45
- Failed logins | 3
- Privilege escalation | 12 

@cpascale43,
As suggested, we will remove the following visuals:
All pie charts and bar charts

And add the following visual:

Aggregated Security Events Over Time – This will display the breakdown of each category over time. It will be created using a custom field derived within the pipeline by bifurcating events based on predicates.

We couldn’t create the Security Event Types visual as suggested, since we’re unable to break down subcategories within categories. For example, under the authentication category, subcategories like “successful logins” or “failed logins” aren’t available, as such details are not present in the logs.

this should be owned by @nfritts's team, the input is owned by their team and also system unified logs. Any reason we are linked here?

good catch @narph agreed, + @marc-gr as he has been assisting with development here

we probably want to use 9.2.0 since is the latest

do we want this or multiple data streams instead? are there any advantages to this approach vs the other?

Since the logs are coming from a single source and only differ by filters such as authentication, user, or network, a single data stream is sufficient. We usually create separate streams only when data originates from different endpoints.

We also have many cases were datastreams are more of a logical separation, eg mimecast, google_workspace, postgresql, etc., in this case I think it could have some benefits by simplifying the logic in the pipelines by quite a lot, since we will always know the event types, and we can reduce logic to identify them. Would make for cleaner pipelines, easier to maintain in the future, and less prone to breaking if anything changes. Is this something we could consider?

Just a feedback, even if the data comes from a single source, if the context/dataset is different like authentication, user and network, they should be stored in different data streams because they may have different requirements for retention, a different volumetry and also require different custom processing.

Having all the data on a single data stream already cause some issues for other integrations like Fortigate and Palo Alto, since this integration can generate an insane amount of logs, this would cause the same problems where the user may want to store some events for a longer time, but is unable because it is all in the same data stream.

For example, in our cause we would have a different retention for authentication and user events and another for network, this would not be possible with a single data stream.

Agreed @leandrojmp , we are working on separating the events into different data streams, soon will update the PR.

same here, the owner should be the @elastic/sec-windows-platform team, @marc-gr is @elastic/sec-linux-platform taking over this one?

should be @elastic/sec-windows-platform

shouldnt this match the level? in this case error

Removing event.type from advanced_monitoring pipeline, as we haven't mapped event.category.
So matching isn't needed here.

could beinteresting to have this as the default value, and let the user override them instead of making them hardcoded here.

We have added option for user to add their predicates as well.
Do you mean to remove the hardcoded ones?

would make sense to standardize this value

we might want to sanitize this and remove the quotes

This has the same predicate as the authentication datastream, how does that work? Does it ingest the data twice?

I can see why splitting it up into different datastreams was discussed earlier and I agree as it would be too big otherwise, but if they have the same predicate then maybe it wouldn't be as useful?

Removing it from user_and_account_management, it should be a part of authentication, thanks.

Any reason we need to store this sort of information twice?

First it is stored in custom mapping, secondly in ecs field., Same for below two fields as well.

Same here

And a third one?

This is not supposed to be an array, its a single keyword representing the most related user of the event, I see related.user is also there, which is where it should end up if its more than 1.

If you are setting the type and the value yourself then its most likely not external? Unsure if that might overwrite it.

The main reason for using external is to retrieve definitions from the ECS. Specifying a type and value will override those fields, but the description will still be sourced from the external reference.

Do we really want to double map all these ECS fields to both have the custom data and the ECS fields?

It is common practice to have both mapped in all integrations, hence followed the same here as well.

Is this really the only way we could parse this? I see that its quite unstructured but it seems at least some of this could maybe be put into a custom pattern and reused for many of these?

It was like 1 in a few million but I did recently observe the connect_duration_ms value being larger then a valid int. Would it be worth updating the durations here to be long?

Replacing int with long, this will handle the above scenario.